Adaptive event timeline in consumer image collections

ABSTRACT

A method for organizing an event timeline for a digital image collection, includes using a processor for detecting events in the digital image collection and each event&#39;s associated timespan; determining the detected events that are significant in the digital image collection; and organizing the event timeline so that the event timeline shows the significant events and a clustered representation of the other events, made available to the user at different time granularities. The organized event timeline is also used for selecting images for generating output.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a Continuation of U.S. application Ser. No.12/710,666, filed Feb. 23, 2010, incorporated herein by reference in itsentirety.

FIELD

The disclosure relates generally to the field of digital imageprocessing, and in particular to a method for organizing groups ofdigital images for efficient management and retrieval of events inconsumer image collections.

BACKGROUND

The proliferation of digital cameras, camera phones, and scanners haslead to an explosion of digital images and videos, creating largepersonal multimedia databases. Since taking digital pictures is easy andpractically free, consumers no longer restrict picture-taking toimportant events and special occasions. Images are being capturedfrequently, and of day-to-day occurrences in the consumers' life. Sincea typical user has already accumulated many years of digital images andvideos, browsing the collection to find images and videos taken duringparticular events is a very time-consuming process for the consumer.

There has been work in grouping images into events. U.S. Pat. No.6,606,411 and U.S. Pat. No. 6,351,556 disclose algorithms for clusteringimage content by temporal events and sub-events. The above two patentsteach how to cluster images and videos in a digital image collectioninto temporal events and sub-events. The terms “event” and “sub-event”are used in an objective sense to indicate the products of a computermediated procedure that attempts to match a user's subjectiveperceptions of specific occurrences (corresponding to events) anddivisions of those occurrences (corresponding to sub-events). Anothermethod of automatically organizing images into events is disclosed inU.S. Pat. No. 6,915,011. The events detected are chronologically orderedin a timeline from earliest to latest.

Using the above methods, the amount of browsing required by the user tolocate a particular event can be reduced by viewing representatives ofthe events along a timeline, instead of each image thumbnail. However, atypical user can still generate over 100 of such events for each year,and more prolific picture-takers can easily exceed a few hundreddetected events per year. There is still a need to create an overallevent timeline structure that is adaptable to the user's picture takingbehavior as well as the time granularity of the events in theircollections, and the ability to select a sub-set of events thatsummarizes the overall collection or a given time period in thecollection.

SUMMARY

In accordance with the present disclosure, there is provided a methodfor organizing an event timeline for a digital image collectioncomprising, using a processor for:

(a) detecting events in the digital image collection and each event'sassociated timespan;

(b) determining the detected events that are significant in the digitalimage collection; and

(c) organizing the event timeline so that the detected events on theevent timeline between the detected significant events are grouped intoclusters on the event timeline.

Since the number of images and videos in a typical user's collection isgrowing rapidly and each user might have a different picture-takingbehavior, there is a critical need to provide the user efficient accessand retrieval of images and important events from their collections. Inthis disclosure, a user's picture-taking behavior in terms of detectedevents is translated into a timeline, where there is a data point foreach time step. Then an adaptive event-based timeline is created whereboth the significance of the detected events as well as the timegranularity of the events are taken into account.

The organization and retrieval of images and videos is a problem for thetypical consumer. It is useful for a user to be able to easily accessand browse an overview of events in their image collection. Technologydisclosed in prior art permits the classification of images in acollection into events, but not the ability to adaptively create anevent timeline that can be tailored to individual users according totheir picture-taking behaviors. As a result, there is a lack ofeffective mechanisms for finding and accessing events that are ofimportance to the users. This disclosure creates an effectiveevent-based timeline at different time scales, where both thesignificance of the detected events as well as the time granularity ofthe events is taken into account. This organized event timeline can beused for selecting images and determining emphasis when creating outputssuch as albums and slideshows from a collection of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system that practices the presentdisclosure;

FIG. 2 is an overall flowchart of the method practiced by FIG. 1;

FIG. 3 is a more detailed flowchart showing one embodiment of the methodof FIG. 2;

FIG. 4 is a more detailed flowchart of a second embodiment of the methodof FIG. 2;

FIG. 5 shows a specific example of the input and output of the eventtimeline organizer 120 of FIG. 2;

FIG. 6 shows a specific example of events generated at differentgranularity settings 215;

FIG. 7 is a specific example of the input and output of the eventtimeline organizer 320 of FIG. 4; and

FIG. 8 is a flowchart of the use of the organized event timeline foroutput creation.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present disclosure can be implemented in computer systems as will bewell known to those skilled in the art. In the following description,some embodiments of the present disclosure will be described as softwareprograms. Those skilled in the art will readily recognize that theequivalent of such a method can also be constructed as hardware orsoftware within the scope of the disclosure.

Because image manipulation algorithms and systems are well known, thepresent description will be directed in particular to algorithms andsystems forming part of, or cooperating more directly with, the methodin accordance with the present disclosure. Other aspects of suchalgorithms and systems, and hardware or software for producing andotherwise processing the image signals involved therewith, notspecifically shown or described herein can be selected from suchsystems, algorithms, components, and elements known in the art. Giventhe description as set forth in the following specification, allsoftware implementation thereof is conventional and within the ordinaryskill in such arts. Videos in a collection are included in the term“images” in the rest of the description.

The present disclosure can be implemented in computer hardware andcomputerized equipment. For example, the method can be performed in adigital camera, a multimedia smart phone, a digital printer, on aninternet server, on a kiosk, and on a personal computer. Referring toFIG. 1, there is illustrated a computer system for implementing thepresent disclosure. Although the computer system is shown for thepurpose of illustrating a preferred embodiment, the present disclosureis not limited to the computer system shown, but can be used on anyelectronic processing system such as found in digital cameras, homecomputers, kiosks, or any other system for the processing of digitalimages. A computer 10 includes a microprocessor-based unit 20 (alsoreferred to herein as a processor) for receiving and processing softwareprograms and for performing other processing functions. A memory unit 30stores user-supplied and computer-generated data which can be accessedby the processor 20 when running a computer program. A display device(such as a monitor) 70 is electrically connected to the computer 10 fordisplaying information and data associated with the software, e.g., byuse of a graphical user interface. A keyboard 60 is also connected tothe computer. As an alternative to using the keyboard 60 for input, amouse can be used for moving a selector on the display device 70 and forselecting an item on which the selector overlays, as is well known inthe art. Input devices 50 such as compact disks (CD) and DVDs can beinserted into the computer 10 for inputting the software programs andother information to the computer 10 and the processor 20. Stillfurther, the computer 10 can be programmed, as is well known in the art,for storing the software program internally. In addition, media files(such as images, music and videos) can be transferred to the memory 30of the computer 10 by use of input devices 50 such as memory cards,thumb drives, CDs and DVDs, or by connecting a capture device (such ascamera, cell phone, video recorder) directly to the computer 10 as aninput device. The computer 10 can have a network connection 80, such asa telephone line or wireless connection, to an external network, such asa local area network or the Internet. Software programs and media filescan be transferred to the computer 10 from other computers or theInternet through the network connection 80.

It should also be noted that the present disclosure can be implementedin a combination of software or hardware and is not limited to deviceswhich are physically connected or located within the same physicallocation. One or more of the devices illustrated in FIG. 1 can belocated remotely and can be connected via a network. One or more of thedevices can be connected wirelessly, such as by a radio-frequency link,either directly or via a network.

Referring to FIG. 2, a user's digital image collection 105 resides inthe memory 30 of the computer 10. The other blocks in the figure areimplemented by a software program and are executed by the processor 20of the computer 10. The digital image collection 105 is provided to anevent detector 110 and a significant event detector 115. The eventdetector 115 groups a user's digital image collection 105 into an eventrepresentation. In the preferred embodiment, the digital imagecollection 105 is grouped into temporal events and sub-events usingevent and sub-event detectors described in U.S. Pat. No. 6,606,411 andU.S. Pat. No. 6,351,556. The above two patents teach how to clusterimages and videos in a digital image collection into temporal events andsub-events. The terms “event” and “sub-event” are used in an objectivesense to indicate the products of a computer mediated procedure thatattempts to match a user's subjective perceptions of specificoccurrences (corresponding to events) and divisions of those occurrences(corresponding to sub-events). Briefly summarized, a collection ofimages is classified into events by determining a threshold for timedifferences between adjacent images of the collection based on atwo-mean clustering of a histogram of time differences between adjacentimages over the entire collection. The images are separated into groupscorresponding to events based on having time differences at the boundarythat are higher than the threshold determined in the earlier step. Foreach event, sub-events can be determined (if any) by comparing the colorhistogram information of successive images as described in U.S. Pat. No.6,351,556. This is accomplished by dividing an image into a number ofblocks and then computing the color histogram for each of the blocks. Ablock-based histogram correlation procedure is used as described in U.S.Pat. No. 6,351,556 to detect sub-event boundaries. Another method ofautomatically organizing images into events is disclosed in U.S. Pat.No. 6,915,011. Briefly summarized, according to one aspect of the abovedisclosure, an event clustering method uses foreground and backgroundsegmentation for clustering images from a group into similar events.Initially, each image is divided into a plurality of blocks, therebyproviding block-based images. Using a block-by-block comparison, eachblock-based image is segmented into a plurality of regions including atleast a foreground and a background. One or more luminosity, color,position or size features are extracted from the regions and theextracted features are used to estimate and compare the similarity ofthe regions including the foreground and background in successive imagesin the group. Then, a measure of the total similarity between successiveimages is computed, thereby providing image distance between successiveimages, and event groups are delimited from the image distances. Theevent timespan is the time between the beginning and end of the event,where the beginning of the event is represented by the capture time ofthe first image in the event, and the end is represented by the capturetime of the last image of the event (or the end of capture of the videosegment if the event ends with a video clip).

Referring to FIG. 2, the significant event detector 115 identifiessignificant events in the digital image collection 105. The digitalimage collection 105 is provided to the significant event detector 115.The detector uses the image capture date and time information extractedfrom the EXIF metadata stored in the image files of the digital imagecollection 105 by the capture device (such as a camera) to generate atime-series that shows the number of images captured on each calendarday of the time period covered by the digital image collection 105. Thistime-series is analyzed to find a model that fits the series. There aremany well established methods for time-series modeling (“Introduction toTime Series and Forecasting”, Brockwell and Davis, Springer-Verlag2002). The model that is appropriate in this situation is the ARIMA(Auto-Regressive Integrated Moving Average) model (Brockwell and Davis,supra, pp 179-187). The model ARIMA (p,d,q) has three main parameters—pbeing the order of the autoregressive component, q being the order ofthe moving average component and d being the order of differencingrequired for dealing with the deviations from stationarity. Significantevents in the collection are identified as occurring on the calendardays where the residuals between the fitted model and the actualtime-series are higher than a specified threshold i.e. where the fittedmodel is less able to predict the actual data.

Referring to FIG. 2, an event timeline organizer 120 accepts as inputthe events detected by the digital image collection 105 and thesignificant events identified by the significant event detector 115. Inone embodiment, the event timeline organizer 120 retains the significantevents on the timeline, and groups the events located between adjacentsignificant events on the timeline into clusters called meta-events.Each event occupies a section of the timeline specified by its timespan.As an example, the top row of FIG. 5 shows a timeline with events andsignificant events marked on it. The event timeline organizer producesthe output shown in the bottom row of FIG. 5, where events between thesignificant events have been clustered into meta-events. A resultingorganized event timeline 150 has a reduced number of top-level groupingsthat include meta-events (that are clusters of events) and significantevents on the timeline. The user is therefore, presented with a lesscluttered top-level view, while still keeping potential events ofinterest (the significant events) accessible at the top-level view.Other events can be accessed by expanding the meta-events, which can bedescribed by the time durations they cover (for example, “Feb. 20-Apr.2, 2009”).

The method of U.S. Pat. No. 6,606,411 also provides a few key parametersfor adjusting the number of events and sub-events generated from anygiven collection of images. The main parameter, “granularity”,determines the overall granularity of event clustering. Valid values arein the range [−1, 1] with 0 being the “optimal” tuned setting. Largervalues increase the number of events found by the algorithm whilesmaller values decrease the number of events. The actual internalmechanism has to do with adjusting the various thresholds for thetwo-means clustering. Some of these thresholds include high and lowthresholds for global histogram comparisons and high and low thresholdsfor multi-block histogram comparisons to determine whether to merge twoevents or two sub-events. The events detected continue to bechronologically ordered in a timeline from earliest to latest. FIG. 6 isan illustration of the set of events generated using different settingsof the granularity parameter. The graph plots the number of eventsdetected during the span of one year. In the example shown, the finalnumber of events for the entire year varies from less than 10 to closeto 170 based on the granularity parameter setting provided.

Referring to FIG. 3, the digital image collection 105 is processed byevent detectors 215 with the granularity parameter set at a range ofvalues covering the [−1, 1] range. The digital image collection 105 isalso processed by the significant event detector 115 and the output isprovided to an event timeline organizer 320. In this embodiment of theevent timeline organizer 320, the timeline is broken up into timeperiods around significant events. The boundaries of the time periodsare at the mid-point of the time between any two adjacent significantevents. In each time period containing a significant event, the eventsdetected at the smallest granularity setting (producing fewest events)is selected that still contains the significant event as a detectedevent. For example, referring to FIG. 7, the top three timelines showthe significant event detector 115 output at different granularities(granularity 1 being the smallest, and granularity 3 being the largest).The significant events are the highlighted events. The last row shows anorganized event timeline 350 with different time periods of thetime-line selecting events from different granularity settings, based onthe smallest granularity that provides a correct representation of thesignificant event in that time period. This method ensures thatsignificant events are available on the organized timeline 350 and otherevents are presented at appropriate granularity around the significantevents.

Referring to FIG. 4, the digital image collection 105 is processed byevent detectors 215 with the granularity parameter set at a range ofvalues covering the [−1, 1] range. The digital image collection 105 isalso processed by the significant event detector 115 using time bins ofdifferent granularities. A set of time units are determined—these timeunits can be a few months (capturing a season), a month, a week, a day,or hours—the size of the unit being referred to as granularity. Thetime-series at different granularities is produced by counting thenumber of images captured in the given time unit or bin. The output ofthe event detectors 215 and the significant event detector 115 atdifferent granularities 415 is provided to a time granularity selector425, along with additional inputs 420. The time granularity selector 425selects a time granularity and extracts the set of events andsignificant events corresponding to that time granularity from its inputsets. The additional inputs 420 can include user actions, systemrequirements or user preferences. The set of events and the significantevents selected are passed on to the event timeline organizer 320described earlier to produce an organized timeline 450. The eventtimeline organizer 120 can also be used instead of event timelineorganizer 320 to produce a different organized timeline for each timegranularity. In one embodiment of the disclosure, the user can thenselect the timeline from this set of event timelines. The additionalinputs 420 are used as follows. In a browsing application, events andsignificant events can be selected at the time granularity at which theuser selects to view the collection. For example, if the user is viewinga short time-span of a few weeks, a finer granularity setting (highernumber) is used for selecting the events and significant events;whereas, if the user is viewing a longer time-span (e.g. the collectionover five years), a lower value of granularity is selected for eventsand significant events. The system requirement in terms of displaycapability can also dictate the number of top-level events on theorganized event timeline, and therefore, the granularity selected. Forexample, if approximately 10 events will fit the display, then thegranularity is selected so that the number of events and significantevents is close to that number. The user can also set the preference forviewing the events at a certain granularity, and the time granularityselector selects this setting for the given user.

One of the challenges of creating an output album (hard copy) orslideshow (soft copy) from a collection of images is the selection ofimages to include and the emphasis to be placed on each of the selectedimages. Referring to FIG. 8, the organized event timeline 810 (which canbe 150, 350 or 450) is provided to a media selection module 820 whichselects the images to be used from the digital image collection 105. Themedia selection module 820 also receives the output specifications 815as input. The output specifications 815 include the number of images tobe selected, and the section of the timeline to be used for theselection e.g. select 12 images from a one year time-span to generate apicture calendar as output. The media selection module 820 prioritizesevents in the time-span selected as follows. The events are weighted inproportion to the number of images in the event, after inflating thenumber of images in significant events by multiplying by a significancefactor (e.g. 2.0). The number of images in the output specification isdistributed among the events proportional to their weight. Let . . . bethe image counts of m events, where

n=(significance factor*actual image count) for significant eventsand n=actual image count for the rest of the (non-significant) events.Then the weight assigned to any event k is

$\frac{n_{k}}{\sum\limits_{i = 1}^{m}\; n_{i}},$

and the number of images assigned to this event is the (weight*number ofimages specified by the output specifications). Note that fractionalnumber of images assigned will need to be rounded to whole numbers, andtherefore, some events can receive 0 images assigned to them (if theyare assigned less than 0.5 images).

The selection of a given number of images from a specific event can bedone simply based on sampling the event images at equal intervals. Thereare also more intelligent methods for performing this selection—takinginto account image quality, diversity and specific semantic featuressuch as people and objects present—which can be used in this step. Forexample, the selection can be based on the technical quality of theimage (A. Loui, M. Wood, A. Scalise, and J. Birkelund, “Multidimensionalimage value assessment and rating for automated albuming and retrieval,”Proc. IEEE Intern. Conf. on Image Processing (ICIP), San Diego, Calif.,Oct. 12-15, 2008) or aesthetic quality of an image (C. Cerosaletti, andA. Loui, “Measuring the perceived aesthetic quality of photographicimages,” Proc. 1st International Workshop on Quality of MultimediaExperience (QoMEX), San Diego, Calif., Jul. 29-31, 2009). However, thisis not a focus of the present disclosure which aims to provide aselection and weighting mechanism at the event-level.

The selected images are provided to an emphasis operator 825. Theemphasis operator 825 generates emphasis scores to highlight particularimages from the selected set. In one embodiment, images fromnon-significant events have the default emphasis score, whereas theselected images that belong to significant events are assigned emphasisscores by multiplying the default score by a significance factor greaterthan 1.0. The emphasis scores can be based on the score for thetechnical quality of the image (: A. Loui, M. Wood, A. Scalise, and J.Birkelund, “Multidimensional image value assessment and rating forautomated albuming and retrieval,” Proc. IEEE Intern. Conf. on ImageProcessing (ICIP), San Diego, Calif., Oct. 12-15, 2008) or aestheticquality of an image (C. Cerosaletti, and A. Loui, “Measuring theperceived aesthetic quality of photographic images,” Proc. 1stInternational Workshop on Quality of Multimedia Experience (QoMEX), SanDiego, Calif., Jul. 29-31, 2009), or a combination of the two scores.The output creation module 850 uses the emphasis score and imageselection to produce a creative presentation of the images. For hardcopy (such as photo-books, calendars, cards), the emphasis score cancontrol the size of the image in the output or the prominence ofposition in the page. For soft copy (such as a slideshow on digitalphoto frames, computer monitor or other display devices), the emphasisscore can be used to assign screen time for the image i.e. images withhigher emphasis scores will be displayed for longer duration. Higheremphasis scores can also provide triggers for highlighting such asdecorative boxes around the image in both forms of presentation.

This disclosure has been described in detail with particular referenceto certain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the disclosure.

PARTS LIST

-   10 Computer-   20 Processor-   30 Memory-   50 Input device-   60 Keyboard-   70 Display device-   80 Network connection-   105 Digital image collection-   110 Event detector-   115 Significant event detector-   120 Event timeline organizer-   150 Organized event timeline-   215 Event detectors at different granularities-   320 Event timeline organizer-   350 Organized event timeline-   415 Significant event detector at different granularities-   420 Additional inputs-   425 Time granularity selector-   450 Organized event timeline-   810 Organized event timeline-   815 Output specifications-   820 Media selection module-   825 Emphasis operator-   850 Output creation module

What is claimed is:
 1. A method comprising: detecting, by a computingdevice, a plurality of events in a plurality of digital images and anassociated timespan for each event of the plurality of events;generating a time series that indicates a number of images capturedduring a time period covered by the digital image collection;identifying a set of events by analyzing the time series usingtime-series modeling; and organizing, by the computing device, theplurality of events into an event timeline, such that detected events onan event timeline between the determined set of events are grouped intoclusters on the event timeline.
 2. The method of claim 1, wherein thedetecting the plurality of events comprises detecting the plurality ofevents based on a selected time granularity.
 3. The method of claim 2,wherein selecting the granularity of event detection is based onchoosing the fewest detected events that preserve the significant eventsduring a given time period.
 4. The method of claim 1, further comprisingenabling user browsing of images associated with at least one detectedevent of the plurality of events.
 5. The method of claim 1, furthercomprising: presenting a display of the event timeline; and receiving auser selection of the event timeline.
 6. The method of claim 4, furthercomprising enabling user browsing of images associated with at least onedetected event of the plurality of events for the selected first eventtimeline.
 7. The method of claim 1, further comprising: producing anemphasis score using an emphasis operator; and controlling apresentation of an output of images based on the emphasis score.
 8. Themethod of claim 1, further comprising analyzing the time series to finda model that identifies the significant events.
 9. The method of claim8, wherein the identifying comprises determining when the model is lessable to predict actual data of the time series.
 10. A system comprising:a memory; and a processor coupled to the memory, wherein the processoris configured to: detect a plurality of events in a plurality of digitalimages and an associated timespan for each event of the plurality ofevents; generate a time series that indicates a number of imagescaptured during a time period covered by the digital image collection;identify a set of events by analyzing the time series using time-seriesmodeling; and organize the plurality of events into an event timeline,such that detected events on an event timeline between the determinedset of events are grouped into clusters on the event timeline.
 11. Thesystem of claim 10, wherein the detection of the plurality of eventscomprises a detection of the plurality of events based on a selectedtime granularity.
 12. The system of claim 11, wherein the selection ofthe granularity of event detection is based on choosing the fewestdetected events that preserve the significant events during a given timeperiod.
 13. The system of claim 10, wherein the processor is furtherconfigured to enable user browsing of images associated with at leastone detected event of the plurality of events.
 14. The system of claim10, wherein the processor is further configured to: present a display ofthe event timeline; and receive a user selection of the event timeline.15. The system of claim 10, wherein the processor is further configuredto: produce an emphasis score using an emphasis operator; and control apresentation of an output of images based on the emphasis score.
 16. Anon-transitory computer-readable medium having instructions storedthereon that, upon execution by a computing device, cause the computingdevice to perform operations comprising: detecting a plurality of eventsin a plurality of digital images and an associated timespan for eachevent of the plurality of events; generating a time series thatindicates a number of images captured during a time period covered bythe digital image collection; identifying a set of events by analyzingthe time series using time-series modeling; and organizing the pluralityof events into an event timeline, such that detected events on an eventtimeline between the determined set of events are grouped into clusterson the event timeline.
 17. The computer-readable medium of claim 16,wherein the detecting the plurality of events comprises detecting theplurality of events based on a selected time granularity.
 18. Thecomputer-readable medium of claim 17, wherein selecting the granularityof event detection is based on choosing the fewest detected events thatpreserve the significant events during a given time period.
 19. Thecomputer-readable medium of claim 16, further comprising enabling userbrowsing of images associated with at least one detected event of theplurality of events.
 20. The computer-readable medium of claim 16,further comprising: presenting a display of the event timeline; andreceiving a user selection of the event timeline.